The Zombie keyboard: Working on the Economy app for iPad was a lot of fun and it also presented its fair share of challenges. One of these challenges related to an annoying zombie keyboard bug. This bug made it impossible to dismiss the keyboard using approved/public methods/APIs. This meant that the keyboard/keypad could be hanging on to the screen even after the relevant text view control was released and no longer displayed. So users would be looking at a zombie keypad that wasn’t connected to any text object.
Killing the zombie with a private API: It was (late night) 3:30am when I realized that the zombie keypad issue was a bug in the iPad SDK and not in my code. At that point, I should have probably called it a day, but didn’t. As I investigated the bug further, I found that the (dismissKeyboard) private API took down the keyboard perfectly. With exception handlers and checks for respondsToSelector, the code seemed reasonably safe (though – in general – I agree with Apple’s reasoning that private APIs shouldn’t be used)
App-store rejection: After a 9-day review cycle, we got a very politely worded email stating that the app had been rejected because of its usage of private APIs. Since the private APIs were being used to work-around a bug in the public API, there seemed to be a reasonable chance that Apple might make an exception. So I filed an appeal with Apple’s app-review board.
An impasse: Apple’s review team did a great job in following up with me (email and phone calls) and we discussed the matter in detail. However, I couldn’t find out the approximate timeframe within which the app-review-board might make their Yes/No decision on the appeal. The appeal seemed to present a difficult problem. On one hand, the private APIs were used because of an SDK bug, on the other hand, approving the private APIs would have set a bad precedent. In the meanwhile, the app was no closer to being released. That is when I decided to email Steve Jobs.
“Ram, this is Steve”: A couple of hours after I sent the email, I was at a noisy soccer (for kids) arena when I heard my iPhone ring. The caller-id, the caller saying “Ram, this is Steve” and that he was calling from Apple did suggest that it could really be Steve Jobs. He confirmed it when I asked.
Steve Jobs’ decision: Steve Jobs talked to me about my appeal and the “no private APIs” policy. I then asked him whether an exception could be made to the policy because the private APIs were necessitated by the SDK bug. He reiterated his point and I accepted it. I told him that I would replace the modal form sheet with alternative UI. Removing the form sheet (and therefore its bug) would eliminate the need for the private API.
A conscientious guy: Steve Jobs has a well-deserved reputation for creating great quality products and for his passion for excellence and user experience. I’ve also read that he is a detail-oriented executive and a hands-on guy who is intimately involved with his company’s work (in a way that few other CEOs are).
His phone-call reinforced those notions and went further to suggest that he was also a very conscientious guy who cared about people. The fact that he took the time to read my email, think about the app and then personally call me was amazing.
Upshot: I resubmitted the ‘Economy for iPad’ app after removing private APIs. The app uses Federal data to give you a snapshot of the US economy. The app review board and app review team graciously expedited the review of the app and it is now available for download at the app store. We have been releasing iOS apps since 2008 (the year when the app store was launched) and this one has the most fascinating submission story among all our apps. I’m now looking forward to seeing how the app does in the iPad app store.
Cascade Software Corporation 
In essence, the story is that the app developer initially violated the App Store prohibition against private APIs (for a good reason) and his app was rejected. Frustrated with the appeals process (it is not clear after how long), the app developer decided to contact SJ. After a return call from SJ, the app developer modified his UI to avoid the buggy Apple code so that the private API could be eliminated, thus complying with the App Store rules. His app was subsequently accepted. That summarizes all of the pertinent issues in the story, which actually isn’t much of a story at all.
I like the hands on approach of SJ. But I believe that it is setting a very poor precedent to encourage people to contact SJ about anything and everything, potentially bypassing perfectly good processes. There is no evidence in this story that the app developer gave the appeals process a fair shake or pursued other avenues to address the iOS bug.
The main item of interest in my opinion is the Apple software bug that leads to hanging of the keyboard on the display. Is there an effective avenue for developers to report bugs in iOS? What is it? How is that bug being handled? Are bugs only squashed at formal iOS releases? If so, we had better hope that iOS is very clean code compared to most commercial software.
Apple has a pretty good bug reporting system. I had filed this bug in Apple’s bug-reporter system. I’ve also filed other bugs. In general, I get an email response within 24 hours (either resolving the issue as a known problem or asking for additional details etc…)
Btw as described (in the post), we did go through the standard appeal process (see paragraph on “impasse”).
[Nov 13 Update to address other points]
1. In general, a good CEO is likely to have a better judgment of what is a good use of their time and it would a bit presumptuous for outsiders to assert that the CEO isn’t using his/her time wisely.
2. As Mehul explains below, this particular case wasn’t a run-of-the-mill use of private api and it fell into a gray area.
3. The workaround works fine, but as HN reader jws said “he went with his second choice for a user interface that is probably not quite as good”
Apple bug tracking is the PITS!! I could not (and still cannot) connect the White Apple wireless keyboard via BT to my iPad, I filed a bug, send the log files as requested about 3 months ago. Still No response. If Apple products do not work with each other, which products will?
I don’t think this sets a bad precedent at all. Ram sent an email to SJ, maybe out of frustration that the bug report he submitted was either not acknowledged or resolved. Use of private APIs when the public APIs in the SDK have a bug is somewhat of a grey area. In my opinion what Ram did was not a flagrant violation of the developed rules.
What I am impressed with is the speed with which SJ responded to the email. Which CEO we know does that?
bugreport.apple.com
Great story!
It’s great to hear of Steve’s personal time and involvement this case.
Is there any word on whether or not the bug is fixed in the latest iOS 4.2 SDK?
Neat story. Apple’s review folks are really quite helpful. It’s easier to get an app approved these days b/c of policy changes. Very awesome that Steve called you directly.
Interesting post – not what I was expecting when I read ‘zombie keyboard’. I agree with Apple in this regard, and your analysis – they couldn’t let you use a private API just to work around a bug, due to the precedent that would set. I’m glad you were level-headed enough to understand the business considerations rather than posting an invective against what a control freak Apple is.
Nice post.
So, Apple knew that your program was using a private API? Is every program submitted to App store is submitted with an source code or what?
they do static analysis of the binary.
They don’t need the source; they can analyze the binary (in essence, partially reverse-engineer your app) to determine what offsets (and thus symbols) you linked against in the Frameworks.
No. But they have a battery of tests which look for (amongst other things) calls to private APIs.
APIs are simply entry points to the operating system or library they are part of. It is trivial to get a list of invoked APIs if you have access to the OS/library source or debug builds, which Apple does. You can often determine the calls purely from the application binary by examining the addresses/offsets specified on out-of-process function calls.
In short, no, Apple doesn’t receive your source code for iOS App Store submissions. They don’t need it.
According to the Developer Forums, the fact the keyboard does not dismiss when you call resignFirstResponder on a form sheet modal view “is not a bug”:
https://devforums.apple.com/message/166801#166801 (requires login)
The full response from an Apple employee: “Was your view by any chance presented with the UIModalPresentationFormSheet style? To avoid frequent in-and-out animations, the keyboard will sometimes remain on-screen even when there is no first responder. This is not a bug.”
It looks like you’ve misinterpreted the “developer forum” message.
In any case, this is an (iPad SDK) bug and resignFirstResponder should dismiss the keyboard.
I had filed the bug in Apple’s bug-reporter system. Apple Technical support has also discussed the bug with me.
just a thought, but wouldn’t a good outcome be if they fixed the public API?
A bug report was filed and you can assume the bug will be fixed in a future version. This is not Windows we’re talking about. There have been something like 25 releases of iOS over its roughly 40 month life.
This does not surprise me that Steve would be so hands-on. its nice to see that he took the time out of his busy day to contact you, that little human touch is what Apple is all about.
@Linux: it’s my understanding that Apple has developed a tool that runs on the submitted binaries to detect calls to unauthorized APIs. With due respect to your nom, this shouldn’t be surprising. Using /usr/bin/strings as a sample binary, try:
$ ldd /usr/bin/strings
which will list all of the dynamic libraries which the ‘strings’ program links against, and
$ objdump -R /usr/bin/strings
which lists all of the APIs used by ‘strings’. All that’s left is to look through the output for unapproved libraries/APIs. No source code required.
@Linux: “Is every program submitted… with source code?”
Of course not. ‘API’ by definition means the program will make calls to existing OS frameworks (say, ‘Edit text field’ which automatically calls for ‘Display the keyboard’), so any ‘unapproved’ calls (like calling the system’s keyboard directly) will easily be caught by ‘eavesdropping’ on the calls the app makes during testing. I believe many external calls can also be found listed by name in the object code, when viewed in a hex editor. Just my 0.02.
I’m glad to hear that the zombie keyboard is an iOS, as i ran into this as well and had to design my own new UI layout to avoid it’s use. However i was using an iPod running iOS 4.1
This was an ill-defined borderline area; involving Steve for a policy decision was, IMHO, appropriate; but now we (the dev. community) knows where that line is… thanks for the post. 🙂
@Ram
What solution did you end up going with. I’m in the same boat and was rejected too. I don’t have the time to redo this portion of the UI. It’s a glaring bug and doesn’t appear to be fixed in 4.2 either.
I understand your point about not having time to “to redo this portion of the UI”. Unfortunately, you don’t have any other options.
Your best solution would probably be to re-implement your UI code to not use UIModalPresentationPageFormSheet. You could either create a custom view or use a different type of UIKit object
Nice article.
How did you solve the zombie keyboard problem in modal form sheet? I’m having the same problem.
See this
The solution is to resize a UIModalPresentationPageSheet immediately after presenting it. See my answer at stackoverflow.com/questions/3372333
I haven’t verified the “solution”, however, at best, this seems like a hack.
I’m sure, this behavior isn’t documented by Apple and so the “solution” could stop working in the next version of the iPad OS.
In any case, it is possible that Apple may fix the bug in iOS 5.0 and that might eliminate the need for the hacks.
As of iOS 4.2, this bug still exists and this fix still works. As always, test it as you would all your code in the next iOS beta. Unofficial workarounds are sometimes the only solutions we have. If you must dismiss the keyboard in a partial-screen modal, and you can’t change your UI, this is the only way I’ve found to do it.
Hi — what email address did you use to contest the app rejection? Did you simply reply to the rejection letter?
Any help would be appreciated.
You can contact the App Review Board and appeal against rejections through http://developer.apple.com/appstore/resources/approval/contact.html